During my internship with VMware, I worked in the Horizon team in the EUC business unit. For my intern project, I worked on test automation for the Horizon Agents Installer, or HAI. For those who may not know, Horizon Agent is needed for virtual machines accessed by Horizon Client, and HAI installs various selectable features for these VMs.

Since Horizon Agent could be installed by many kinds of VMs, developers must test that HAI works on different OSes and that any new changes do not cause regressions. However, since there wasn’t any automated testing before, developers had to test the installer manually on multiple VMs, which can be time-consuming and repetitive, or the tests were simply incomplete or not done.

Our goals for this project were to automate tests for installing, repairing, uninstalling, and upgrading. We also wanted to allow multiple developers to run their tests simultaneously, and to design the test so that infrastructure management is minimized. To accomplish these goals, we wanted to use PowerShell scripts as it would be easy to perform tasks needed for tests, such as silently installing the Agent, checking if the features were installed properly, and writing log files. We wanted to use Azure for our test infrastructure to create and store test resources, such as our test VMs and installer files. If we were to use vSphere, VMware’s cloud computing platform, instead for our infrastructure, our scale would have been limited to pre-configured settings. By using Azure’s public cloud, we have essentially infinite capacity for the number of test VMs and concurrent tests. We wanted to use Jenkins to schedule daily tests and send notifications of the test results to quickly address any bugs created during development. Lastly, we wanted to have sufficient logs to allow developers to easily see what issues there are if tests fail, and they could start working on fixes quicker.

Before running the tests, users need to use scripts to set up their Azure account, and create a resource group containing shared images. The images are created using VMs of Windows 7, 10, and 2019. The shared images are used as a “template” for the VMs and could be used by multiple developers when running the tests. However, users could also customize their source images if they would like specific settings or programs on the test VMs.

To run the tests locally, developers would simply start one script. The script first queries Azure for a resource group containing the shared images, then uses those images to create the test VMs in a new resource group. Other resources are also created, such as storage containers to store the HAI builds, and file shares to store the HAI logs after the tests.

One issue that we had to resolve was downloading the HAI builds onto the test VMs. Of course, if we had only wanted to test released builds, it would be very easy to do this. However, we wanted to be able to test the latest builds still in development and sandbox builds. We also wanted to be able to customize the branch, version, and build number. These builds would have to be downloaded from BuildWeb with VMware VPN access. As such, a script would download the requested build on the local machine (which would have VPN access) through the REST API, and the build would be uploaded as a blob to the storage containers. By doing this, the test VMs could access the builds with the blob download link.

After uploading all of the HAI builds that would be used in the tests, test scripts would be run remotely by pushing Powershell script executions to the test VMs. To have more modularity, each set of tests are separated into different scripts, which would run the individual tests on each VM. By doing this, it would be more convenient to customize which sets of tests to run. For each test, the result for each VM would be printed to the console in real-time. A log file would also record the steps taken and the outcome of the test, including the reason if it fails. These test log files would be stored on the local machine in the TEMP directory. After each set of tests, HAI logs would be zipped and uploaded to the file share, and a link to download the HAI logs would be printed out to the console. By having substantial logging, developers could easily find when and why the test failed, so they can quickly start working on a fix.

The sets of tests were:

• install: silently install, repair, and uninstall the latest build.
• upgrade: silently install an older version, upgrade to the latest build, and uninstall.
• UI: perform various checks on the UI. Some of the things we checked were the features shown, the silent install command, error windows, and rollback after canceling installs.

After the tests finished, the console would print out the result of each test for each VM again, along with the file path containing the test logs.

For any scheduled job from Jenkins, the test will be run from a slave VM, and the results will be sent to developers’ emails or Slack after the test suite is finished. The log files can be attached to the emails and are also easily accessible from the Jenkins dashboard. The daily Jenkins test would tell the developers if something has gone wrong recently. Individual developers can also use this test setup from their local machines for their personal sandbox testing.

While I was still working on the test automation, I had to create a poster and presentation about this project for VMware’s intern poster session, where all R&D had to present their projects. Although it occurred in the middle of my internship, I had finished most of the test automation, and only had not worked on the set of UI tests yet. I presented my project again at the end of my internship to a couple of related teams.

In the future, the current tests can act as a framework for new tests for new functionality to the HAI. It would be great to have more customization on the tests, as currently, developers would have to manually edit the code to run only some tests. A good way to do this would be to create a desktop application for running the tests or perhaps adding more parameters to the Jenkins test suite.

I believe that this project was a very good first experience working in a real professional setting. I was able to learn how to go through the code review process using ReviewBoard, and we had daily meetings reporting on our progress, which we tracked using JIRA sprint boards. It was slightly different from the conventional internship since everything was remote, but this only emphasized the need for clear communication with my mentor and manager. I enjoyed being a part of developing a widely used program, and I want to give special thanks to my mentor Joel and my manager Sonia.